Visual sensor coordinate system setting jig and setting method

ABSTRACT

A visual sensor coordinate system setting jig comprising a jig including a plurality of feature points arranged at known intervals in an array state corresponding to a visual sensor coordinate system, and a visual sensor to receive an image of the plurality of feature points of the jig, at least three of the plurality of feature points each having a distinguishing appearance which the visual sensor recognizes to discriminate the at least three feature points from the other of the plurality of feature points, the at least three feature points representing an origin and coordinate axes of the visual sensor coordinate system, and the visual sensor to identify the origin and the coordinate axes of the visual sensor coordinate system based on the array state of the plurality of feature points and the distinguishing appearance of the at least three feature points.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a coordinate system setting jig for avisual sensor used in detecting the position and attitude of an objectin controlling an automatic machine such as a robot, for example, andalso to a sensor coordinate system setting method using the jig.

2. Description of the Related Art

A visual sensor, which includes a camera having a CCD element array andthe like and an image processing device for storing and analyzing animage picked up by means of the camera, is often used for the control ofan automatic machine, such as a robot, or other similar applications.Conventionally known is a system arranged as follows. For example, inperforming welding, transportation, and other operations for theworkpiece by using the robot, a deviation of a workpiece as an object ofoperation, from its reference position or reference attitude is detectedby means of one or a plurality of visual sensors, the data indicative ofsuch deviation is transmitted as correction data for the robot positionor attitude to a robot controller, and the position or attitude iscorrected in accordance with the correction data as the robot executes ataught operation.

Such a system must be arranged so that a coordinate system (cameracoordinate system, more specifically CCD pixel array, for example) onwhich the image data for the object of operation recognized by eachvisual sensor is based and a coordinate system (usually work coordinatesystem) set in the robot can be integrated, and position data (pixelvalue data) detected based on the former coordinate system by theindividual visual sensors can be converted into data (work coordinatesystem data) represented by the latter coordinate system.

Usually, therefore, a sensor coordinate system, which can be recognizedin common by the visual sensors and the robot controller, is establishedso that the pixel value data detected by the visual sensors aretemporarily converted into data on the sensor coordinate system, and theresulting data are further converted into data on the workpiececoordinate system on which actual robot drive control is based. (In somecases, the sensor coordinate system may be made to be identical with thework coordinate system.)

As can be inferred from the above example of application to the controlof the robot, in constructing a system intended for utilizing forvarious purposes the object position information detected by the visualsensors, it is usually necessary for such detected information to bearranged for the conversion into objective data. More specifically, theinformation, such as the pixel value data, which can only be recognizeddirectly by the visual sensors, need to be arranged so that they can beconverted into the information or the data which can be represented byan objective coordinate system. In the description to follow, theobjective coordinate system will be referred to generally as the sensorcoordinate system without regard to the application.

In order to convert the pixel value data into the data represented bythe predetermined sensor coordinate system, it is sufficient to have thesensor coordinate system recognized by the visual sensors in any way,utilizing the CPU, image processor, memories, etc. included in thevisual sensor so that the correlation between the sensor coordinatesystem and the pixel value can be stored in the visual sensor. Afterthis coordinate system is established once, the data represented on thesensor coordinate system can be obtained with respect to the objectrecognized by the visual sensor, in accordance with the aforesaidcorrespondence relationship stored in the visual sensor, unless thesensor coordinate system is changed. Even in case the sensor coordinatesystem is changed, the sensor coordinate system set in the visual sensorcan be changed by utilizing a conversion program as long as therelationship between new and old sensor coordinate systems is known.

Conventionally, in a common method for setting such sensor coordinatesystem, a sensor coordinate system setting jig with three feature pointsthereon is located in a predetermined position (e.g., fixed position ona fixed plane in a work space), the three feature points, whosepositions on the sensor coordinate system are known, are caught in thefield of view of the camera of the visual sensor, the pixel value datafor the individual feature points are stored; and, based on these dataand the known position data, the visual sensor recognizes the origin andthe directions and scales (actual size of one pixel on the plane of theobject, or reciprocal of scale factor on the pixel plane) of individualcoordinate axes (normally X- and Y-axes) of the sensor coordinatesystem.

There is another method in which the feature points are increased tofour or more, e.g., seven, in number, in consideration of that object isrecognized through the lens system of the camera of the visual sensor,these seven feature points are recognized by the visual sensor, and thesensor coordinate system is established in the same processings. (Tsai'scamera model indicates that at least seven feature points are neededwhen the lens system is used.)

According to the prior art described above, all of the three or morefeature points must be recognized by means of the visual sensor incatching a plurality of feature points on the jig in the field of viewof the camera of the visual sensor and executing the sensor coordinatesystem setting operation.

If the feature points are as few as three or thereabout, it isrelatively easy to catch all the feature points in the camera view fieldof the visual sensor and coordinate the pixel data corresponding theretoto the data of the individual feature points on the sensor coordinatesystem. Since the lens system has aberration, it is inevitably difficultto set the sensor coordinate system accurately in consideration of imagedistortion which is liable to occur in the peripheral area of the fieldof view, in particular.

This drawback can be avoided by increasing the feature points so thatthey are distributed throughout the field of view of the camera. If thisis done, however, another problem such that all the feature pointscannot always be contained in the field of view will be encountered. Forexample, if an application to the robot control is assumed, for example,the actual size of the field of view of the visual sensor willinevitably have to be changed correspondingly when the size or operationarea of the object of operation or required position detecting accuracyis changed.

Thus, in the case of high-accuracy position detection for smallworkpieces locally confined to a narrow area, the camera is located in arelatively close position, so that the actual size of the field of viewis accordingly small. Thus it is necessary to prepare a small-sized jighaving feature points arranged with a high array density in a narrowarea.

Consequently, in the case of relatively rough position detection forlarge workpieces scattered in a wide area, the camera is located at alarger distance, so that the actual size of the field of view isaccordingly wide. To meet this, therefore, it is desirable to prepare alarge-sized jig having feature points arranged with a low array densityin a wide area.

Thus, according to the prior art system, jigs of various sizes must beprepared depending on the nature of the operation, and besides, thekinds of programs to be provided for coordinate setting increasecorrespondingly, and the reusability of the programs lowers.

Furthermore, coupled with the previously discussed problem of the sizeof the field of view, when any of the feature points cannot berecognized satisfactorily, general state of the array of the featurepoints cannot be covered easily. Thus, it is difficult to recognize theposition of the origin and the directions of the coordinate axes, andthis entails a problem such that the coordinate setting itself cannot beexecuted normally.

SUMMARY OF THE INVENTION

The object of the present invention is to get over the above drawbacksof the prior art, and to provide a novel sensor coordinate systemsetting jig, which can be used in common despite variations in theactual size of the field of view depending on the nature of operation,and can set a sensor coordinate system by a relatively simple operationnot necessarily requiring all of feature points to be recognized.

Further, the present invention is intended to provide a sensorcoordinate system setting method which utilizes the novel jig describedabove.

In order to achieve the above objects, the present invention provides avisual sensor coordinate system setting jig which comprises a number offeature points (marks having a narrower extent than the latticeinterval; the shape or pattern of the marks is optional and not limitedto the so-called dot pattern, the same applies hereinafter) arranged atknown intervals in individual lattice point positions of a lattice (theshape of each unit lattice is not limited to a square configuration; toindicate any unit having a regular array shall be called latticehereinafter) corresponding to a visual sensor coordinate system andcapable of being recognized by a visual sensor, a relatively smallnumber of feature points, at least three, among the number of featurepoints, having an additional feature by which the feature points can bediscriminated from the remaining feature points by means of the visualsensor, and the relatively small number of feature points, at leastthree, representing the origin and coordinate axes of the visual sensorcoordinate system by an array state identifiable of being identified bythe visual sensor or a feature of the feature points themselves, andfurthermore the present invention contrives a visual sensor coordinatesystem setting method which comprises a step of locating the jig inposition; a step of observing a number of feature points on the visualsensor coordinate system setting jig by the visual sensor andrecognizing the origin and coordinate axes of the visual sensorcoordinate system in accordance with the array form of the relativelysmall number of feature points, at least three, or the feature of thefeature points themselves; and a step of recognizing the correspondingpositions of the individual lattice points on the visual sensorcoordinate system in accordance with the contents of the recognition.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a conceptual diagram showing an arrangement used in setting asensor coordinate system in a visual sensor by means of a jig accordingto the present invention,

FIG. 2(a) is a diagram showing a feature point array according to oneembodiment of the jig of the present invention;

FIG. 2(b) is a diagram showing the state of an image obtained byshooting the array by a camera;

FIG. 3 is a diagram showing a feature point array according to anotherembodiment of the jig of the present invention;

FIG. 4 is a block diagram showing the principal part of an example of animage processing device used in carrying out a sensor coordinate settingmethod using the jig of the present invention;

FIG. 5 shows the first half of a flow chart illustrating an example of aprocessing for carrying out the sensor coordinate system setting methodof the present invention; and

FIG. 6 shows the second half of the flow chart illustrating the exampleof the process for carrying out the sensor coordinate system settingmethod of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring first to examples shown in FIGS. 1 to 3, the principle, onwhich a feature point array provided on a jig according to the presentinvention is recognized by a visual sensor and utilized for establishingthe sensor coordinate setting will be described.

FIG. 1 shows the way a jig J, in the form of a thin plate located in apredetermined position on a work plane G (one fixed plane in a workspace for the case a three-dimensional operation is assumed), is shot bya camera C, which is connected to an image processing device 10, toestablish a sensor coordinate system. The optical axis of the camera Cis inclined at an angle to the work plane G and hence, to the surface ofthe jig, and the camera catches in its visual range Γ a region whichincludes the central portion of the feature point array of the jig J.

Various versions of the feature point array of the jig J may becontrived. For example operation will be described for the case in whichthe one shown in FIG. 2(a) is used. FIG. 1 shows only some of featurepoints. The feature points of the jig J are arranged at known intervalsd in the form of a lattice. Among these feature points, P₀₀ P₂₀ and P₀₁are given an additional feature (here large black dots or large-areablack dots) by which they can be discriminated from the other featurepoints (small black dots or marks). A straight line from the featurepoint P₀₀ to P₂₀ is given as the X-axis, and a straight line from thefeature point P₀₀ to P₀₁ as the Y-axis. A coordinate system having thisP₀₀ as its origin is supposed to be established as the sensor coordinatesystem in the visual sensor.

Viewing the whole feature point array, feature points P_(mn) (m=-5, -4,. . . , 6, 7; n=-7, -6, . . . , 6, 7) are provided on lattice pointswhich are situated at distances md in the X-axis direction and distancesnd in the Y-axis direction from the origin P₀₀. In other words, thecoordinate values of the feature points P_(mn) are given by [md, nd].

FIG. 2(b) shows the way the feature point array is imaged on a pixelplane of the camera C or a screen for displaying the same. Since thecamera C is inclined at an angle to the work plane G (feature pointarray plane), a lattice formed by a map {Q_(mn) } of the feature pointarray {P_(mn) } is distorted from a set of squares, corresponding to thedirection of inclination (direction of a normal vector from the pixelplane). In the drawing, Q_(mn) represents map points for the featurepoints P_(mn). Thus, the large-black-dot feature points P₀₀, P₂₀ and P₀₁are mapped on Q₀₀, Q₂₀ and Q₀₁, respectively. V_(X) and V_(y)conceptually represent vanishing points of individually mapped lines oflattice.

The feature points P₀₀ , P₂₀ and P₀₁ having the additional feature aregiven a conspicuous feature which can be easily discriminated by meansof the image processing device 10 of the visual sensor. Since these fewadditional feature points are located within a relatively narrow rangearound the origin, such as the central portion of the jig, it is tocontain all of them (normally within three or three plus several innumber) securely in the camera view field.

The jig or the feature point array thereof is designed relatively largerto prepare for the case that a relatively large area can be covered withthe field of view. Thus, the camera field of view is unable to cover allthe feature points even in the case shown in FIG. 2. More specifically,although P₅₅, P₆₀, etc. are contained in the camera view field, P₇₇,P₋₅₋₇, etc. are outside the camera view field. It is a remarkablefeature of the present invention that the sensor coordinate system canbe set without any hindrance at all even in such a situation.

The following is a description of the principle on which the coordinatesetting is executed in this precondition. First, the visual sensor ismade to recognize the images Q₀₀, Q₂₀ and Q₀₁ of the three additionalfeature points P₀₀, P₂₀ and P₀₁, and is given, as an instruction, thecorrespondence of the respective (central) pixel values I₀₀ =(α, β), I₂₀=(α, β) and I=(α, β) of Q, Q and Q₀₁ to the respective coordinate values[0, 0], [2d, 0] and [0, d] of P₀₀, P₂₀ and P₀₁, Here (α, β) indicatesthat a pixel corresponding to (the center of) Q_(ij) is the pixel in theα 'th row and α 'th column of a matrix which a pixel array forms. Theadditional feature points P₀₀, P₂₀ and P₀₁ can be discriminated by theother feature points by only causing the visual sensor to extract thefeature points which have the additional feature (large area in thiscase).

If this is done, then the visual sensor can be considered to havefetched minimum necessary information for the determination of theorigin and the directions and scales of the X- and Y-axes of the sensorcoordinate system. The X- and Y-axes can definitely be discriminated bya feature of the array, P₀₁ -P₂₀ distance>P₀₀ -P₂₀ distance>P₀₀ -P₀₁distance. The method for extracting the additional feature points is notlimited to the method which utilizes the aforesaid relationships of thedistances between the points. For example, the distinctions between Q₀₀,Q₂₀ and Q₀₁ may be taught to the visual sensor so that the origin andthe directions and scales of the X- and Y-axes of the sensor coordinatesystem can be determined thereby, in accordance with information thatthe three points Q₀₀, Q₂₀ and Q₀₁ form a triangle and that angles Q₂₀Q₀₀ Q₀₁, Q₀₀ Q₀₁ Q₂₀ and Q₀₀ Q₂₀ Q₀₁ are approximately 90°, 60° and 30°,respectively, on condition that the camera is not inclined extremely.Also, the condition, angle Q₂₀ Q₀₀ Q₀₁ >angle Q₀₀ Q₀₁ Q₂₀ >angle Q₀₀ Q₂₀Q₀₁, may be used.

When the teaching of the correspondence of the pixel values I₀₀, I₂₀ andI₀₁ to the respective coordinate values [0, 0], [2d, 0] and [0, d] ofP₀₀, P₂₀ and P₀₁ is completed in this manner, estimated values I'_(mn)=(α', β') of the pixel values of Q_(mn) corresponding to all of residuallattice points can be obtained in accordance with data for therespective pixel values I₀₀ =(α, β), I=(α, β) and I₀₁ =(α, β) of thethree points.

Measured data of the pixel values of Q_(mn) corresponding to all theremaining lattice points are seized by the visual sensor as far as therecognition of the feature points has not failed, except only thatprovided the correspondence between the individual data and m and n isnot determine. It is necessary, therefore, only that the measured dataand m and n be coordinated by utilizing the aforesaid estimated values.

In consideration of the possibility of wrong recognition of the featurepoints, however, abnormal measured values can be excluded by setting asuitable allowable value Δ or by other means, or otherwise, theestimated values can be settled by utilizing an interpolation methodbased on measured values corresponding to other reliable adjacentlattice points.

Based on the principle described above, m and n can be coordinated withrespect to all feature point images {Q_(ij) }.

Naturally, it is to be understood that the method for determining theindividual pixel values I_(ij) for all the feature point images {Q_(ij)} is not limited to the method described above. For example, it isevident that the same result can also be obtained by a method such thatpixel values, such as Q₁₀, Q₃₀, Q₄₀, Q₅₀, Q₀₂, Q₀₃, Q₀₄, Q₀₅, etc., onthe images of the X- and Y-axes are first detected from the observedvalues and settled, and selection and settlement of the most approximateobserved value, among remaining observed values for the nearest pointsQ, are repeated in succession.

If a state is attained such that m and n are settled for all the featurepoint images {Q_(ij) } is attained, the coordinate values [md, nd] ofP_(ij) on the sensor coordinate system are coordinated with theindividual pixel values I_(mn), whereupon setting the sensor coordinatesystem is substantially completed.

That is, the pixel value data for (the center of) an image obtained whena specific point (e.g., center of a workpiece) of an optional object isobserved can be converted into data on the sensor coordinate system inaccordance with a suitable conventional computation, such as theinterpolation method based on the pixel value data for the individuallattice points.

If I_(mn) is obtained, the respective positions of the vanishing pointsV_(X) and V_(y) on the pixel plane can be computed, so that thedirection (directional cosine) of the optical axis of the camera canalso be computed on the basis of the computation result. In detectingthe three-dimensional position of the object by means of a plurality ofvisual sensors, this information on the direction of the optical axis isutilized.

The above is a description of the function or the principle ofcoordinate setting for the case in which the jig shown in FIG. 2(a) isused. Alternatively, however, a jig may be designed either for providingvaried additional features or for locating no additional feature pointon the origin. FIG. 3 shows an example of this arrangement.

The following is a brief description of a feature point array of the jigshown in FIG. 3. There are two types of additional feature points, largeblack dots and large white circles. The sensor coordinate system is seton condition that point of intersection of a segment connecting a largewhite circle H₋₁₀ and a large black dot H₁₀ and a segment connecting alarge black dot H₀₋₁ and a large black dot H₀₁ represents the originH₀₀, a straight line extending from the large white circle H₋₁₀ to thelarge black dot H₁₀ represents the X-axis, and a straight line directedfrom the large black dot H₀₋₁ to the large black dot H₀₁ represents theY-axis. It is to be understood without any special explanation that thesensor coordinate system can be set in accordance with the sameprinciple as the one described in connection with the case of FIG. 2.

Referring additionally to FIGS. 4, 5 and 6, an example of a procedurefor setting coordinates by a sensor coordinate system setting methodaccording to the present invention will now be described. According tothis procedure, the jig J of FIG. 2(a), as an example of the sensorcoordinate system setting jig according to the present invention, isarranged in the manner shown in FIG. 1, and it is shot by means of thecamera of the visual sensor.

FIG. 4 is a block diagram illustrating the camera C, which constitutesthe visual sensor used to carry out the present invention, and theprincipal part of the image processing device 10 connected thereto. Inthe drawing, the image processing device 10 includes a centralprocessing unit (hereinafter referred to as CPU) 11. The CPU 1 isconnected, by means of a bus 21, with a camera interface 12, imageprocessor 13, console interface 4, communication interface 15, TVmonitor interface 16, frame memory 17, memory 18 for control softwareformed of a ROM, program memory 19 formed of a RAM, and data memory 20formed of a nonvolatile RAM. The camera interface 12 is connected withthe camera C which catches the jig J in its view field Γ in the mannershown in FIG. 1. Several such cameras C are connected as required, andare arranged so as to detect the position of an object of working (e.g.,workpiece to be machined) on the work plane G after the sensorcoordinate system is set, as well as to transmit data for positioncorrection to a robot which is connected to the communication interface15.

The several cameras C, though only one of which is illustrated,sometimes may be connected simultaneously to the communication interface15 for the operation. In this case, for example, a common sensorcoordinate system is set for the individual cameras.

An image picked up by the camera C is converted into a gradational imagewhich is converted into a gray scale, and is loaded into the framememory 17. The image processor 13 serves to process the image datastored in the frame memory 17, recognize the images Q_(mn) of thefeature points P_(mn) of the jig J, and detect their positions(corresponding pixel values; central pixel values if the images have anextent).

The console interface 14 is connected with a console 23, which includesvarious command keys and an input unit for performing operations such asentry, editing, registration, execution, etc. of application programs,besides an output unit. The output unit can display, for example, listsof menus and programs used to set various data.

The memory 18 for control software is loaded with a control program withwhich the CPU 11 controls the visual sensor system. The program memory19 is loaded with programs which are created by a user.

The communication interface 15 is connected with an automatic machine(robot in this case) or the like which utilizes position information onthe work or the like detected by the visual sensor. On the other hand,the TV monitor interface 16 is connected with a TV monitor 24 which canselectively display the image being shot by means of the camera C or theimage stored in the frame memory 17.

Although the above-described arrangement is basically the same as thearrangement of a conventional visual sensor system, it differs from theconventional system arrangement in that the memory 18 for controlsoftware or the data memory 20 is loaded with programs and data (S₀, d,etc.), which are necessary for the execution of processings such as theones shown in the flow charts of FIGS. 5 and 6, in carrying out thepresent invention.

If the aforesaid sensor coordinate setting programs are supposed to havealready been registered and stored in the memory 18 for controlsoftware, a preliminary operation or first thing to do for setting thesensor coordinate system is to set the cameras C to be used in the imageprocessing device 10. For example, this setting is achieved by assigningconnector numbers in the camera interface 12, when setting coordinatesystems are set for selected ones of the cameras which are connected tothe camera interface.

When the setting the cameras C is finished, the data for the jig J arestored in the data memory 20. When using the jig according to thepresent invention, particularly important input data are those datawhich for discriminating the distance d between the lattice points onwhich the feature points are arranged, from the additional feature.

In the case where several types of jigs (e.g., two 10 types shown inFIGS. 2 and 3) are used properly or in the case where two- orthree-dimensional sensor coordinate setting is selected correspondingly,codes for specifying the types of the jigs and coordinate setting andcodes and data for specifying algorithms necessary for recognizing theorigin and coordinate axes and settling coordinate values correspondingto the individual lattice point images in accordance with the images ofthe additional feature points are inputted or assigned.

If the data for the jig are necessary for the operation of the robot,such requirement can be met by inputting such data to a memory in arobot controller (not shown) through the communication interface 15.

Referring now to the flow charts of FIGS. 5 and 6, described below is anexample of a processing operation of the image processing device 10, inwhich the sensor coordinate system setting is executed by using the jigJ according to the present invention shown in FIG. 2, on the basis ofthe aforesaid preparations.

First, the coordinate setting operation is started when an operatorgives a sensor coordinate setting start command from the console 23. Inselecting the type of the jig J and the type of the contents of sensorcoordinate setting, codes for specifying the types of the jig andcoordinate setting and codes for specifying the algorithms are assigned,and the specified codes and necessary incidental data are loaded for thedata memory 20 (START). Immediately after the start, a feature pointimage number index q and an additional feature point number index r areadjusted for being q=r=0 (Step S1), whereupon pickup the image by thecamera C is 10 started. An image being shot by means of the camera C isfetched by the frame memory 17, the feature point images in the viewfield is recognized by the image processing device 10 (Step S2), numbers1, 2, . . . , N are assigned to the feature point images in properorder, and the images, along with corresponding pixel values, are storedin the data memory 20 (Step S3).

When the storage of the feature point data is finished, 1 is added tothe feature point number index q (Step S4). In a first processing, q=1is obtained. Then, a feature point image Q₁ with a first number ischecked to see if its image is a large black dot in terms of whether ornot its area S_(q) is greater than the aforesaid set value S_(O) (StepS5). The area of each image can be determined by counting pixels whichconstitute an image Q_(q). If the image is a large black dot, 1 is addedto the additional feature point number index r (Step S6), and theresulting value is then identified and stored as Q_(q) =R_(r), that is,data of an r'th additional feature point (Step S7).

If Q_(q) is not a large black dot image, the program returns to Step S4,whereupon 1 is added again to the feature point number index q, and theprogram advances to Step S5. Before all of three large black dot imagesR₁, R₂ and R₃, equal in number to the large black dots (additionalfeature points) of FIG. 2(a) are detected the decision in Step S8 nextto Step S7 is NO, and so after the program returns again to Step S4,Steps S4 to S8 are repeated.

When all the three large black dot images R₁, R₂ and R₃ are detected,the decision in Step S8 is YES, whereupon central pixel values I(R₁),I(R₂) and I(R₃) of the large black dot images R₁, R₂ and R₃ arecomputed, and the result is loaded into the data memory 20 (Step S10).In this stage, the correspondent relationship between I(R₁), I(R₂) andI(R₃) and P₀₀, P₂₀ and P₀₁ is not recognized.

Using the aforesaid data I(R₁), I(R₂) and I(R₃), therefore, a distanceD₁₂ between I(R₁) and I(R₂) on the pixel plane, distance D₂₃ betweenI(R₂) and I(R₃) on the pixel plane, and distance D₃₁ between I(R₃) andI(R₁) on the pixel plane are computed individually (Step S10). In theorder of the magnitude of these distances, I(R₁), I(R₂) and I(R₃) aremade to correspond equally to P₀₀, P₂₀ and P₀₁ on 1-to-1 basis, and theindividual pixel values are determined as I₀₀, I₂₀ and I₀₁ (Step S11).

For example, if D₁₂ >D₂₃ >D₃₁ is given, for example, R₃ is an image ofP₀₀ on the origin, R₂ is an image of the point P₂₀, and R₁ is an imageof P₀₁. Thus, relationships, I(R₁)=I₀₁, I(R₂)=I₂₀, and I(R₃)=I₀₀, areestablished. Likewise, if D₂₃ >D₁₂ >D₃₁, relationships, I(R₁)=I₀₀,I(R₂)=I₂₀, and I(R₃)=I₀₁, are established (see FIG. 5 for the abovesteps).

When the recognition of the additional feature points is completed inthis manner, the estimated values I'_(mn) of the pixel values of theresidual feature point images are computed in accordance with therecognition (Step S12: see FIG. 6 from this step on). For thecomputation method, as mentioned in connection with the description ofthe function, it is necessary only to make computations corresponding tothe aforementioned equations (1) and (2) be executed.

Thereupon, the feature point image number index q is restored to 1 (StepS13), and a processing is started to determine the estimated pixelvalues I'_(mn) to which the individual feature point images correspond.

First, it is determined whether or not Q₁ is any of R₁ to R₃ (Step S14).When YES, since the pixel values are already determined in Step S11,they are not the objects of coordination with the estimated pixelvalues, and the program advances to Step S17, whereupon 1 is added tothe feature point number index q. Subsequently, it is checked whether ornot the processing of all of N number of feature points is finished(Step S18). Unless it is finished, the program returns to Step S14.

If the decision in Step S14 is NO, there should be only one estimatedvalue I'_(mn) which is approximate to a measured pixel value I_(q)(central pixel value if there are a plurality of values) of Q_(q). Then,according to equation (3) mentioned in connection with the descriptionof the function, distances, Δq-10, Δq-30, Δq-40, . . . , Δq-77, . . . ,Δq-5-7, between I and the individual estimated values I'_(mn) arecomputed, (m, n) which gives the minimum distance Δq-mn is obtained(Step S15), and the corresponding pixel value I_(q) is determined asI_(mn) (Step S16).

Thus, the aforesaid processings of Steps S13 to S18 are executed for allQ_(q) (q=1, 2, . . . , N) caught in the field of view of the camera C,whereupon the feature points P_(mn) corresponding to the pixel values ofall Q_(q) recognized by the camera are determined.

If this correspondent relationship is established, pixel valuescorresponding to the individual lattice points [md, nd] can be settled,since the coordinate values of the feature points P_(mn) on the sensorcoordinate system are given by lattice point coordinate values [md,rid]. This correspondent relationship, along with necessary incidentaldata, is loaded into the data memory 20 (Step S19) in order to beregistered in the image sensor, whereupon setting the sensor coordinatesystem is finished (END).

That is, after this, the position or region of a plane, at which or inwhich the jig J is placed corresponds to the image recognized by anygiven pixel, can always be computed by the reverse calculation based onthe pixel data of the registered individual lattice points. Thus, bypreviously registering a program for the reverse calculation in thememory 18 for control software or the robot controller, imageinformation detected by the visual sensor can be converted into data onthe sensor coordinate system as an objective coordinate system andutilized for control robot control and the like.

As described above, according to the present invention, the jig isprovided with a few additional feature points which can be discriminatedfrom the other feature points and represent the origin and thedirections and scales of the coordinate axes. Even though the actualsize of the view field varies depending on the nature of the operation,therefore, fundamental data for the sensor coordinate system can easilyand surely be obtained. As far as such few additional feature points canbe recognized, the operator can set the sensor coordinate system by asimple operation without paying attention to whether the other featurepoints are in the view field or whether they are recognizedsuccessfully.

Moreover, even when a considerable portion of the feature points is outof the field ov view, this will not cause any problem so thattroublesome operations, such as preparing jigs of different sizes forthe individual operations and resetting the coordinate system, can beeliminated. Thus, the operating efficiency can be improved largely.

Not only all the feature points need not be recognized, but also all thedata for the recognized feature points can be utilized efficiently asthe data for the connection between the camera coordinate system and thesensor coordinate system, so that the coordinate setting inconsideration of those marginal areas of the camera view field where theimage is liable to be distorted but without substantially caring aboutthe relation between the jig size and the camera view field, therebycontributing to the improvement of the detecting accuracy of the visualsensor.

We claim:
 1. A visual sensor coordinate system setting jig comprising:aplurality of feature points of a jig arranged at known intervals in anarray state corresponding to a visual sensor coordinate system; and avisual sensor to receive an image of said plurality of feature points ofsaid jig, at least three of said plurality of feature points each havinga distinguishing appearance which said visual sensor recognizes todiscriminate said at least three feature points from the other of saidplurality of feature points, said at least three feature points eachhaving said distinguishing appearance representing an origin andcoordinate axes of said visual sensor coordinate system, and said visualsensor to identify said origin and said coordinate axes of said visualsensor coordinate system based on said array state of said plurality offeature points and said at least three feature points each having saiddistinguishing appearance.
 2. A visual sensor coordinate system settingmethod comprising the steps of:locating a plurality of feature points ofa jig arranged at known intervals in an array state corresponding to avisual sensor coordinate system; receiving an image of said plurality offeature points of said jig; discriminating at least three of saidplurality of feature points each having a distinguishing appearance fromthe other of said plurality of feature points, said at least threefeature points each having said distinguishing appearance representingan origin and coordinate axes of said visual sensor coordinate system;and identifying said origin and said coordinate axes of said visualsensor coordinate system based on said array sate of said plurality offeature points and said at least three feature points each having saiddistinguishing appearance.
 3. A visual sensor coordinate system settingmethod according to claim 2, further comprising the step of recognizingthe corresponding positions of said plurality of feature points in saidarray state on said visual sensor coordinate system in accordance withsaid identifying step.